sr: add basic auth support to the schema registry #6639

NyaliaLui · 2022-10-05T15:27:11Z

Cover letter

At Redpanda there is an effort to improve security by supporting HTTP Basic Authentication on some of out services such as the Schema Registry. PR #6452 introduced HTTP Basic Authentication to the Pandaproxy but not the Schema Registry.

This PR adds Basic Auth support to all Schema Registry endpoints but it does not include support for multiple authenticated connections. That will come at a later time.

This PR includes configuration changes to enable basic auth:

The new configuration options to enable basic auth are attached to the listener in schema_registry_api:

authentication_method: string
    Optional values: none, http_basic
    Default value: none

Examples for the new authn config:

schema_registry:
  schema_registry_api:
  - address: localhost
    port: 8081
    authentication_method: none

schema_registry:
  schema_registry_api:
  - address: localhost
    port: 8081
    authentication_method: http_basic

After this PR, the leftover tasks are:

Add kafka client cache to the Pandaproxy endpoints /consumers
Address follow-up points from pp: add basic auth to /brokers #6452
Add ducktape tests for: 1) mTLS only 2) mTLS + Basic Auth
Create upgrade, scale, and feature tests in ducktape
Manual tests on cloud infra

Backport Required

UX changes

none

Release notes

Features

Adds HTTP Basic Auth to all Schema Registry endpoints

src/v/pandaproxy/types.h

BenPope · 2022-10-06T10:39:30Z

src/v/pandaproxy/schema_registry/service.cc

+        if (rq.authn_method == config::rest_authn_method::http_basic) {
+            // Will throw 400 & 401 if auth fails
+            auto auth_result = rq.service().authenticator().authenticate(
+              *rq.req);
+            try {
+                auth_result.require_authenticated();
+            } catch (const ss::httpd::base_exception& ex) {
+                // The failure from require_authenticated() will throw 401
+                // instead of 403. So convert the exception to 403.
+                if (
+                  ex.status() == ss::httpd::reply::status_type::unauthorized) {
+                    throw ss::httpd::base_exception(
+                      ex.what(), ss::httpd::reply::status_type::forbidden);
+                } else {
+                    // Something else went wrong
+                    throw;
+                }
+            }
+        }


This looks familiar, can the code be shared?

Something like (this might be a bit out of date): e1f92a2 and e73d5b7
And then it's possible to add some nice helpers: 7d82be4
Which can simplify the handlers: eccfeed

But I'll leave it to you to figure out the best way to reduce the dupolication.

I like your context server changes but I think I'll put them in #6626 after we get the initial feature PRs merged

src/v/pandaproxy/schema_registry/configuration.cc

src/v/pandaproxy/schema_registry/service.cc

tests/rptest/tests/schema_registry_test.py

NyaliaLui · 2022-10-07T22:21:27Z

Things left todo on this PR:

factor out credential validation
factor out alot of the duplicate code in ducktape tests

dotnwat · 2022-10-07T23:15:51Z

factor out credential validation
factor out alot of the duplicate code in ducktape tests

sounds like something we might potentilally be able to do in follow-up PR(s) but it's up to you.

BenPope

Looks great

src/v/pandaproxy/auth_utils.h

src/v/pandaproxy/schema_registry/service.h

tests/rptest/tests/schema_registry_test.py

dotnwat

looks awesome. just a couple super minor things

src/v/config/rest_authn_endpoint.cc

src/v/pandaproxy/auth_utils.h

tests/rptest/tests/schema_registry_test.py

NyaliaLui · 2022-10-11T13:56:54Z

BLOCKER: TimeQueryTest.test_timequery.batch_cache=True #6685 (comment)

The failure is unrelated to this PR though since the failing test does not use:

Basic Auth
Schema Registry

NyaliaLui · 2022-10-11T15:36:20Z

/ci-repeat 5

NyaliaLui · 2022-10-11T20:34:20Z

Re-running CI now

dotnwat · 2022-10-11T22:20:45Z

I restarted CI. debug run had the TimeQuery issue whose fix is landing soon, and release build ran out of disk space :(

dotnwat · 2022-10-12T03:06:07Z

/ci-repeat 5

This commit tests HTTP Basic Authentication on the Schema Registry endpoint POST compatibility/subjects/{subject}/versions/{version}

This commit tests HTTP Basic Authentication on the Schema Registry endpoint DELETE /subjects/{subject}

This commit tests HTTP Basic Authentication on the Schema Registry endpoint DELETE /subjects/{subject}/versions/{version}

This commit tests HTTP Basic Authentication on the Schema Registry endpoint GET subjects/{subject}/versions/{version}/referencedBy

dotnwat · 2022-10-12T17:10:07Z

CI is green now and the latest force push appears to contain changes that are completely disjoint with this PR

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639 (cherry picked from commit d4bc3c8)

NyaliaLui requested review from dotnwat and BenPope October 5, 2022 15:27

NyaliaLui self-assigned this Oct 5, 2022

BenPope reviewed Oct 6, 2022

View reviewed changes

mmedenjak added kind/enhance New feature or request area/schema-registry Schema Registry service within Redpanda area/security labels Oct 6, 2022

dotnwat reviewed Oct 7, 2022

View reviewed changes

tests/rptest/tests/schema_registry_test.py Show resolved Hide resolved

NyaliaLui force-pushed the basic-auth-sr branch from f1244fb to a6fbe0d Compare October 7, 2022 22:20

github-actions bot added the area/redpanda label Oct 7, 2022

NyaliaLui force-pushed the basic-auth-sr branch from a6fbe0d to 154be2b Compare October 10, 2022 16:53

NyaliaLui marked this pull request as ready for review October 10, 2022 16:54

NyaliaLui requested review from dotnwat and BenPope October 10, 2022 16:54

NyaliaLui force-pushed the basic-auth-sr branch from 154be2b to 63737cc Compare October 10, 2022 16:58

BenPope reviewed Oct 10, 2022

View reviewed changes

src/v/pandaproxy/auth_utils.h Outdated Show resolved Hide resolved

src/v/pandaproxy/schema_registry/service.h Show resolved Hide resolved

tests/rptest/tests/schema_registry_test.py Outdated Show resolved Hide resolved

dotnwat reviewed Oct 10, 2022

View reviewed changes

NyaliaLui mentioned this pull request Oct 10, 2022

pp: add kafka client cache to /consumers routes #6693

Merged

12 tasks

NyaliaLui force-pushed the basic-auth-sr branch from 63737cc to c957334 Compare October 10, 2022 21:25

BenPope previously approved these changes Oct 10, 2022

View reviewed changes

dotnwat previously approved these changes Oct 10, 2022

View reviewed changes

dotnwat mentioned this pull request Oct 11, 2022

Failure in storage_single_thread_rpunit test_concurrent_prefix_truncate_and_gc #6717

Closed

NyaliaLui mentioned this pull request Oct 11, 2022

Failure in NodesDecommissioningTest.test_recommissioning_one_of_decommissioned_nodes #6721

Closed

NyaliaLui added 4 commits October 12, 2022 09:08

sr: compatibility subject version basic auth test

6a29243

This commit tests HTTP Basic Authentication on the Schema Registry endpoint POST compatibility/subjects/{subject}/versions/{version}

sr/tests: add delete subject basic auth test

c6dceca

This commit tests HTTP Basic Authentication on the Schema Registry endpoint DELETE /subjects/{subject}

sr/tests: delete subject version basic auth test

e8e776b

This commit tests HTTP Basic Authentication on the Schema Registry endpoint DELETE /subjects/{subject}/versions/{version}

sr: add subjects referenced-by basic auth test

4154a81

This commit tests HTTP Basic Authentication on the Schema Registry endpoint GET subjects/{subject}/versions/{version}/referencedBy

NyaliaLui dismissed stale reviews from dotnwat and BenPope via 4154a81 October 12, 2022 13:08

NyaliaLui force-pushed the basic-auth-sr branch from c957334 to 4154a81 Compare October 12, 2022 13:08

dotnwat approved these changes Oct 12, 2022

View reviewed changes

dotnwat merged commit 6699047 into redpanda-data:dev Oct 12, 2022

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

97655a0

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta mentioned this pull request Oct 26, 2022

Enable per kafka listener sasl #6940

Merged

6 tasks

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

4d2aecc

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

f637fb1

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

d110ff8

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

e992410

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

f5608aa

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

7ef0df8

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 26, 2022

k8s: Extend schema registry authorization method

c82ff48

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 27, 2022

k8s: Extend schema registry authorization method

1873492

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 27, 2022

k8s: Extend schema registry authorization method

e0572e1

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 27, 2022

k8s: Extend schema registry authorization method

7bee249

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 28, 2022

k8s: Extend schema registry authorization method

730ea51

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

RafalKorepta pushed a commit to RafalKorepta/redpanda that referenced this pull request Oct 30, 2022

k8s: Extend schema registry authorization method

d4bc3c8

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

NyaliaLui deleted the basic-auth-sr branch November 18, 2022 18:19

joejulian pushed a commit to joejulian/redpanda that referenced this pull request Mar 10, 2023

(split) k8s: Extend schema registry authorization method

b2902f2

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

joejulian pushed a commit to joejulian/redpanda that referenced this pull request Mar 24, 2023

operator: k8s: Extend schema registry authorization method

c4e81d0

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639

joejulian pushed a commit to joejulian/redpanda that referenced this pull request Apr 12, 2023

k8s: Extend schema registry authorization method

709da73

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639 (cherry picked from commit d4bc3c8)

joejulian pushed a commit to joejulian/redpanda that referenced this pull request Apr 12, 2023

k8s: Extend schema registry authorization method

569ae99

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639 (cherry picked from commit d4bc3c8)

joejulian pushed a commit to joejulian/redpanda that referenced this pull request Apr 13, 2023

k8s: Extend schema registry authorization method

cecbebc

With changes to authorization configuration options the CRD is updated. REF: redpanda-data#6639 (cherry picked from commit d4bc3c8)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sr: add basic auth support to the schema registry #6639

sr: add basic auth support to the schema registry #6639

NyaliaLui commented Oct 5, 2022 •

edited by andrewhsu

Loading

BenPope Oct 6, 2022

NyaliaLui Oct 7, 2022 •

edited

Loading

NyaliaLui commented Oct 7, 2022

dotnwat commented Oct 7, 2022

BenPope left a comment

dotnwat left a comment

NyaliaLui commented Oct 11, 2022 •

edited

Loading

NyaliaLui commented Oct 11, 2022

NyaliaLui commented Oct 11, 2022

dotnwat commented Oct 11, 2022

dotnwat commented Oct 12, 2022

dotnwat commented Oct 12, 2022

sr: add basic auth support to the schema registry #6639

sr: add basic auth support to the schema registry #6639

Conversation

NyaliaLui commented Oct 5, 2022 • edited by andrewhsu Loading

Cover letter

Backport Required

UX changes

Release notes

Features

BenPope Oct 6, 2022

Choose a reason for hiding this comment

NyaliaLui Oct 7, 2022 • edited Loading

Choose a reason for hiding this comment

NyaliaLui commented Oct 7, 2022

dotnwat commented Oct 7, 2022

BenPope left a comment

Choose a reason for hiding this comment

dotnwat left a comment

Choose a reason for hiding this comment

NyaliaLui commented Oct 11, 2022 • edited Loading

NyaliaLui commented Oct 11, 2022

NyaliaLui commented Oct 11, 2022

dotnwat commented Oct 11, 2022

dotnwat commented Oct 12, 2022

dotnwat commented Oct 12, 2022

NyaliaLui commented Oct 5, 2022 •

edited by andrewhsu

Loading

NyaliaLui Oct 7, 2022 •

edited

Loading

NyaliaLui commented Oct 11, 2022 •

edited

Loading